Skip to content

moe: expert slots in size classes (Flash-Next: 34.4 -> 27.0 GiB pinned for the same slots) - #137

Merged
bong-water-water-bong merged 2 commits into
mainfrom
moe/slot-classes
Sep 26, 2026
Merged

moe: expert slots in size classes (Flash-Next: 34.4 -> 27.0 GiB pinned for the same slots)#137
bong-water-water-bong merged 2 commits into
mainfrom
moe/slot-classes

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

Unsloth's UD quants use bigger types for some layers' experts. With one slot size, the largest, Flash-Next's 9,216 slots pinned 34.4 GiB for 27.5 GiB of expert data. Qwen3-Coder-30B Q4_K_M, which uses Q6_K for some layers' down projection, pinned 8.8 GiB.

ExpertCache now groups layers by the bytes one expert needs (the sum of its aligned read windows). Each group has its own slots and LRU, or one LRU per layer with --per-layer, and every layer keeps the same share of its experts.

Measured on Strix Halo with 1bit moe-cache:

model slots pinned before pinned after
Qwen3.8-Flash-Next UD-Q4_K_XL 9,216 34.4 GiB 27.0 GiB
Qwen3-Coder-30B Q4_K_M 3,072 8.8 GiB 8.2 GiB

A Coder-30B chat replay runs normally: 95.2% hits, 33.9 tok/s at 12.1 ms compute.

The trade-off: the shared LRU no longer moves budget between size classes. Flash-Next has 2 classes, and most models have one.

🤖 Generated with Claude Code

bong-water-water-bong @claude
moe: expert slots in size classes (Flash-Next 9,216 slots: 34.4 -> 27…
714dafe 
….0 GiB pinned)

Mixed-quant (UD) models use bigger types for some layers' experts. One slot size, the
largest, wasted up to 22% of the pinned memory. Layers are now grouped by the bytes one
expert needs; each group has its own slots and LRU, and every layer keeps the same share of
its experts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) September 26, 2026 13:35
@context7

context7 Bot commented Sep 26, 2026 •
edited
Loading

Copy link
Copy Markdown

Docs7 for 1bit-monster/engine

Result Status Action
Deployment ➖ Not used —
Content review ➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan. —

Commit ba4c23f

@github-actions

github-actions Bot commented Sep 26, 2026 •
edited
Loading

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit ba4c23f)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect slot offset calculation

The calculation of off for each slot in ExpertCache::ExpertCache does not correctly account for the size of each slot class. The current code assigns off based on a cumulative sum of lists[li].bytes, but it should be based on the cumulative size of slots within each class, not just the bytes per slot. This could lead to incorrect memory layout and data corruption when accessing expert data.

 
for (int li = 0; li < (int) lists.size(); ++li)
    for (int k = 0; k < lists[li].slots; ++k, ++si) {
        slots_[si].lru_list = li;
        slots_[si].off = off;
        off += lists[li].bytes;
        lru_[li].push_back(si);  // empty slots at the back: taken first
        slots_[si].lru_it = std::prev(lru_[li].end());
    }
Potential LRU list index out of bounds

The code assigns lru_list to slots based on li (the index of the class in lists), but there's no explicit bounds checking to ensure that li is valid. If lists.size() exceeds the size of lru_, this could lead to out-of-bounds access. While the code initializes lru_ with lists.size() elements, it's not clear if this is guaranteed to be safe in all cases.

 
for (int li = 0; li < (int) lists.size(); ++li)
    for (int k = 0; k < lists[li].slots; ++k, ++si) {
        slots_[si].lru_list = li;
        slots_[si].off = off;
        off += lists[li].bytes;
        lru_[li].push_back(si);  // empty slots at the back: taken first
        slots_[si].lru_it = std::prev(lru_[li].end());
    }

@github-actions

github-actions Bot commented Sep 26, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit ba4c23f

@bong-water-water-bong
bong-water-water-bong merged commit f8b3649 into main Sep 26, 2026
5 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the moe/slot-classes branch September 26, 2026 13:39
@github-actions github-actions Bot mentioned this pull request Sep 26, 2026
Merged
@bong-water-water-bong bong-water-water-bong mentioned this pull request Oct 24, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Review effort 4/5

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bong-water-water-bong